Summary

• add terminalbench to the manual run-eval workflow choices
• update the internal run-eval skill docs so agents know the new benchmark option exists
• verify the cross-repo dispatch path against the corresponding evaluation/benchmarks feature branches

Details

• The smoke run used benchmark=terminalbench, eval_limit=5, sdk_ref=main, eval_branch=openhands/terminalbench-ci-490, and benchmarks_branch=openhands/terminalbench-ci-490.
• This PR pairs with matching workflow/input changes in OpenHands/evaluation and benchmark-side Harbor fixes in OpenHands/benchmarks.

Testing

• Triggered workflow dispatch: https://github.com/OpenHands/software-agent-sdk/actions/runs/22823734279
• Confirmed downstream evaluation workflow launch: https://github.com/OpenHands/evaluation/actions/runs/22823745521

Evidence

Verification link: View conversation

Follow-up investigation: the previously cited terminalbench smoke run did not complete end-to-end, so this PR is being moved back to draft pending real live-run evidence.

$ gh run view 22823734279 --repo OpenHands/software-agent-sdk --json status,conclusion,displayTitle,url
{"conclusion":"success","displayTitle":"Run Eval (terminalbench) Smoke test for OpenHands/benchmarks#490","status":"completed","url":"https://github.com/OpenHands/software-agent-sdk/actions/runs/22823734279"}

$ gh run view 22823745521 --repo OpenHands/evaluation --json status,conclusion,displayTitle,url
{"conclusion":"success","displayTitle":"Eval Job (terminalbench) Smoke test for OpenHands/benchmarks#490","status":"completed","url":"https://github.com/OpenHands/evaluation/actions/runs/22823745521"}

$ # Datadog pod logs for eval-22823745521-claude-son* service:python
[2026-03-08 15:11:16 UTC] Benchmark: terminalbench
[2026-03-08 15:11:16 UTC] Dispatching terminalbench build for SDK commit: 77c68ccfd7bdffb27be88e8793f76cafc45faf9d
[2026-03-08 15:11:17 UTC] ERROR: Benchmarks build dispatch failed (status 404): {"message":"Not Found","documentation_url":"https://docs.github.com/rest/actions/workflows#create-a-workflow-dispatch-event","status":"404"}
[2026-03-08 15:11:17 UTC] Deleted temporary branch: dispatch-22823745521

The GitHub Actions runs only proved that the workflow dispatch/deploy path was reachable. Datadog shows the orchestration failed before the evaluation phase, so there was no completed benchmark run, no uploaded results archive, and no Slack success notification.

Likely root cause: OpenHands/evaluation currently derives the benchmark build workflow name as build-{benchmark}-images.yml, which becomes build-terminalbench-images.yml. That workflow file does not exist on OpenHands/benchmarks (including branch openhands/terminalbench-ci-490), so the dispatch returns HTTP 404.

Checklist

• CI passing
• Tests are minimal and pass
• No unnecessary code
• Evidence from live run (with conversation link if available)
• All review comments resolved
• Documentation updated (if applicable)